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Abstract — A Multi- Agent System is a distributed system where 
the agents or nodes perform complex functions that cannot be 
written down in analytic form. Multi-Agent Systems are highly 
connected, and the information they contain is mostly stored in 
the connections. When agents update their state, they take into 
account the state of the other agents, and they have access to those 
states via the connections. There is also external, user-generated 
input into the Multi-Agent System. As so much information is 
stored in the connections, agents are often memory-less. This 
memory-less property, together with the randomness of the 
external input, has allowed us to model Multi-Agent Systems 
using Markov chains. In this paper, we look at Multi-Agent 
Systems that evolve, i.e. the number of agents varies according to 
the fitness of the individual agents. We extend our Markov chain 
model, and define stability. This is the start of a methodology 
to control Multi-Agent Systems. We then build upon this to 
construct an entropy-based definition for the degree of instability 
(entropy of the limit probabilities), which we used to perform 
a stability analysis. We then investigated the stability of evolving 
agent populations through simulation, and show that the results 
are consistent with the original definition of stability in non- 
evolving Multi-Agent Systems, proposed by Chli and De Wilde. 
This paper forms the theoretical basis for the construction of 
Digital Business Ecosystems, and applications have been reported 
elsewhere. 



I. Introduction 

Multi-Agent Systems is a growing field primarily because 
of recent developments of the Internet as a means of circulat- 
ing information, goods and services. Many researchers have 
contributed valuable work to the area in recent years [39]. 
However, despite both Evolutionary Computing and Multi- 
Agent Systems being mature research areas [38], [30] their 
integration, creating evolving agent populations, is a recent 
development [47]. This integration is non-trivial, because 
agents can be considered as state machines and Evolutionary 
Computing algorithms have been developed to work on nu- 
merical data and strings without memory effects. So, our aim 
here is to determine, for Multi-Agent Systems which make 
use of Evolutionary Computing [29], [7], [46], macroscopic 
variables that characterise their stability. 

While there are several definitions of stability defined for 
Multi-Agent Systems [40], [1], [34], [54], [33], they are not 
applicable because of the Evolutionary Computing dynamics 
inherent in the context of evolving agent populations. Chli 
and De Wilde [15] model Multi- Agent Systems as Markov 
chains, which are an established modelling approach in Evo- 
lutionary Computing [44] . They model agent evolution in time 
as Markov processes, and so view a Multi-Agent System 
as a discrete time Markov chain with potentially unknown 
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transition probabilities, considered stable when its state has 
converged to an equilibrium distribution [15]. Also, while there 
is past work on modelling Evolutionary Computing algorithms 
as Markov chains [43], [36], [24], [20], we have found none 
including Multi-Agent Systems. 

Therefore, we decided to extend the existing Chli-DeWilde 
definition of agent stability to include the dynamics of Evolu- 
tionary Computing, including population dynamics and macro- 
states of the population state-space. 

We have applied our efforts reported here to the construction 
of Digital Business Ecosystems [35], [6], Digital Ecosys- 
tems (distributed adaptive open socio-technical systems, with 
properties of self-organisation, scalability and sustainability, 
inspired by natural ecosystems) for conducting business that 
enable network-based economies, levelling the playing field 
for Small and Medium sized Enterprises (SMEs) [5], [10]. 
SMEs provide substantial employment and conduct much 
innovative activity, but struggle in global markets on a far 
from level playing field, where large companies have distinct 
advantages [49]. So, we created Digital Ecosystems to be the 
digital counterparts of natural ecosystems [12], [8], which 
are considered to be robust, self-organising and scalable 
architectures that can automatically solve complex, dynamic 
problems [11], [5], [7]. This lead to a novel optimisation tech- 
nique inspired by natural ecosystems, where the optinusation 
works at two levels: a first optimisation, migration of agents 
which are distributed in a decentralised peer-to-peer network, 
operating continuously in time; this process feeds a second 
optimisation based on evolutionary computing that operates 
locally on single peers and is aimed at finding solutions to 
satisfy locally relevant constraints. So, the local search is 
improved through this twofold process to yield better local 
optima faster, as the distributed optimisation provides prior 
sampling of the search space through computations already 
performed in other peers with similar constraints. Therefore, 
our extended Chli-DeWilde stability was invaluable in un- 
derstanding and developing the evolving agent populations in 
these Ecosystem-Oriented Architectures (EOAs) [9], because 
it was important for us to be able to understand, model, and 
define stability, including the determination of macroscopic 
variables to characterise the stability, of the order constructing 
processes within, the evolving agent populations. 

II. Multi-Agent Systems 

A software agent is a piece of software that acts, for 
a user in a relationship of agency, autonomously in an 
environment to meet its designed objectives [57]. So, a Multi- 
Agent System is a system composed of several software 
agents, collectively capable of reaching goals that are difficult 
to achieve by an individual agent or monolithic system 
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[57]. Examples of problems which are appropriate to Multi- 
Agent Systems research include online trading [42], disaster 
response [45], and modelling social structures [50]. Multi- 
agent systems are applied in the real world to graphical 
applications such as computer games and films. They are also 
used for coordinated defence systems, with other applications 
including transportation, logistics, as well as in many other 
fields. It is widely being advocated for use in networking and 
mobile technologies, to achieve automatic and dynamic load 
balancing, high scalability, and self-healing networks. 

The systems studied by Wiener [56] model information flow 
between an actor and the environment. Input, state and output 
and defined, and consequently an evolution equation can be 
established. In such a system it is possible to distinguish 
an observational sequence (of the inputs), followed by a 
decisional sequence of outputs [53]. 



III. Chli-DeWilde Stability 

We will now briefly introduce Chli-DeWilde stability for 
Multi- Agent System and Evolutionary Computing, sufficiently 
to allow for the derivation of our extensions to Chli-DeWilde 
stability to include Multi-Agent Systems with Evolutionary 
Computing. Chli-DeWilde stability was created to provide a 
clear notion of stability in Multi- Agent Systems [15], because 
while computer scientists often talk about stable or unstable 
systems [52], [4], they did so without having a concrete or 
uniform definition of stability. So, the Chli-DeWilde definition 
of stability for Multi- Agent Systems was created [15], derived 
[14] from the notion of stability defined by De Wilde [19], 
[28], based on the stationary distribution of a stochastic 
system, making use of discrete-time Markov chains, which 
we will now introduced 

If we let / be a countable set, in which each i G / is called 
a state and / is called the state- space. We can then say that 
A = (A^ : i G /) is a measure on I if < Xi < oo for all 
i e I, and additionally a distribution if XI^g/ ^ ^ t^^]. So, 
if X is a random variable taking values in / and we have 
A^ = Pr(X = i), then A is the distribution of X, and we can 
say that a matrix P = {pij : i, j G /) is stochastic if every row 
{Pij : j G /) is a distribution [14], [37]. We can then extend 
familiar notions of matrix and vector multiplication to cover 
a general index set / of potentially infinite size, by defining 
the multiplication of a matrix by a measure as AP, which is 
given by 

{xp), = j2^jPij' (1) 

We can now describe the rules for a Markov chain by a 
definition in terms of the corresponding matrix P [14], [37]. 

Definition 1. We say that (X*)t>o is a Markov chain with 
initial distribution A = (A^ : i G /) and transition matrix 
P = {Pij • hj e I) if' 

^ A more comprehensive introduction to Markov chain theory and stochastic 
processes is available in [37] and [16]. 



1) Pr{X° = io) = and 

2) Pr(X*+i = it+i \ X° =io,...,X' = it)= Pi,u^,. 
We abbreviate these two conditions by saying that (X^)t>o is 
Markov{\, P). 

From this first definition the Markov process is 
memoryless^, resulting in only the current state of the 
system being required to describe its subsequent behaviour. 
We say that a Markov process has a 

stationary distribution if the probability distribution of 
becomes independent of the time t [15]. So, the following 
theorem is an easy consequence of the second condition from 
the first definition. 

Theorem 1. A discrete-time random process (X*)t>o is 
Markov{X, P), if and only if for all t and ig, . . . , it we have 

Ft{X^ =iQ,...,X^ = it) = Xi,Pion ' ••Pit-iif (2) 

This first theorem depicts the structure of a Markov chain 
[14], [37], [16], illustrating the relation with the stochastic 
matrix P. The next Theorem shows how the Markov chain 
evolves in time, again showing the role of the matrix P. 

Theorem 2. Let {X^)t>o be Markov{X,P), then for all 
t,s> 0: 

1) Pr(X^ = j) = {XP^)j and 

2) Pr(X^ = j \ X^ = i) = Pr(X*+^ = j \ X' = i) = 
For convenience {P^)ij can be denoted as pf^. 

Given this second theorem we can define pf^ as the t-step 
transition probability from the state i to j [14], and we can 
now introduce the concept of an invariant distribution [14], in 
which we say that A is invariant if 

AP = A. (3) 

The next theorem will link the existence of an invariant 
distribution, which is an algebraic property of the matrix P, 
with the probabilistic concept of an equilibrium distribution. 
This only applies to a restricted class of Markov chains, 
namely, those with irreducible and aperiodic stochastic 
matrices. However, there is a multitude of analogous results 
for other types of Markov chains which we can refer [37], 
[16], and the following theorem is provided as an indication, 
of the family of theorems that apply. An irreducible matrix 
P is one for which, for all i, j G / there exist a sufficiently 
large t such that pf^ > 0. I matrix P and is aperiodic if for 

^Markov systems with probabilities is a very powerful modelling technique, 
applicable in large variety of scenarios, and it is common to start memoryless, 
in which the output probability distribution only depends on the current input. 
However, there are scenarios in which alternative modelling techniques, like 
queueing systems, are more suitable, such as when there is asynchronous 
communications, and to fully characterise the system state at time (t), the 
history of states at (t-1), (t-2), ... might also need to be considered. 
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all states i G / we have > for all sufficiently large t 
[14], [37], [16]. The meaning of these properties can broadly 
be explained as follows. An irreducible Markov chain is a 
chain where all states intercommunicate. For this to happen, 
there needs to be a non-zero probability to go from any 
state to any other state. This communication can happen 
in any number t of time steps. This leads to the condition 
pfj^ > for all i and j. An aperiodic Markov chain is a chain 
where all states are aperiodic. A state is aperiodic if it is not 
periodic. Finally, a state is periodic if subsequent occupations 
of this state occur at regular multiples of a time interval. 
For this to happen, p\f has to be zero for t an integral 
multiple of a number. This leads to the condition p\f > 
for a-periodicity. For further explanations, please refer to [16]. 

Theorem 3. Let P be irreducible, aperiodic and have an 
invariant distribution, A an arbitrary distribution, and suppose 
that {X^)t>o is Markov{\P) [14], then 

Pr(X* =j)^pfast^oo for all j el (4) 

and 

p\f ^pf as oo for all i, j G /. (5) 

We can now view a system 5* as a countable set of states / 
with implicitly defined transitions between them, and at time 
t the state of the system is the random variable X^, with the 
key assumption that {X%>q is Markov(A, P) [14], [37], [16]. 

Definition 2. The system S is said to be stable when 
the distribution of its states converge to an equilibrium 
distribution, 

Pr(X^ =j)^pfast^oo for allj e I. (6) 

More intuitively, the system S, a stochastic process X^, 
X^, X^, ... is stable if the probability distribution of X^ 
becomes independent of the time index t for large t [15]. 
Most Markov chains with a finite state- space and positive 
transition probabilities are examples of controllable stable 
systems, because after an initialisation period they reach a 
stationary distribution [14]. 

A Multi-Agent System can be viewed as a system S, with 
the system state represented by a finite vector X, having 
dimensions large enough to represent the states of the agents 
in the system. The state vector will consist of one or more 
elements for each agent, and a number of elements to define 
general properties^ of the system state. Hence there are many 
more states of the system (different state vectors) than there 
are agents. We can then model agent death, i.e. not being 
present in the system, by setting the vector elements for that 
agent to a fixed value that is shared by no other agent, called 
d [14]. 

^ These general properties are intended to represent properties that are 
external to the agents, and as such could include the coupling between the 
agents. However, we would expect such properties, as the coupling between 
the agents, to be stored within the agents themselves, and so be part of the 
elements defining the agents. 



IV. Including Evolution 

Having now introduced Chli-DeWilde stability, we will 
now briefly introduce Evolutionary Computing, sufficiently to 
allow for the derivation of our extensions to Chli-DeWilde 
stability to include Multi-Agent Systems with Evolutionary 
Computing. Evolution is the source of many diverse and 
creative solutions to problems in nature [17], [22]. However, 
it can also be useful as a problem- solving tool in artificial 
systems. Computer scientists and other theoreticians realised 
that the selection and mutation mechanisms that appear so 
effective in biological evolution could be abstracted to a 
computational algorithm [30]. This Evolutionary Computing 
is now recognised as a sub-field of artificial intelligence 
(more particularly computational intelligence) that involves 
combinatorial optimisation problems [3]. 

A. Evolutionary Algorithms 

Evolutionary algorithms are based upon several fundamental 
principles from biological evolution, including reproduction, 
mutation, recombination (crossover), natural selection, and 
survival of the fittest. As in biological populations, evolution 
occurs by the repeated application of the above operators 
[2]. An evolutionary algorithm operates on the collection 
of individuals making up a population. An individual, in 
the natural world, is an organism with an associated fitness 
[27]. So, candidate solutions to an optimisation problem 
play the role of individuals in a population, and a cost 
(fitness) function determines the environment within which 
the solutions live, analogous to the way the environment 
selects for the fittest individuals. The number of individuals 
varies between different implementations and may also vary 
during the use of an evolutionary algorithm. Each individual 
possesses some characteristics that are defined through its 
genotype, its genetic composition, which will be passed 
onto the descendants of that individual [2]. Processes of 
mutation (small random changes) and crossover (generation 
of a new genotype by the combination of components from 
two individuals) may occur, resulting in new individuals with 
genotypes differing from the ancestors they will come to 
replace. These processes iterate, modifying the characteristics 
of the population [2]. Which members of the population 
are kept, and used as parents for offspring, depends on 
the fitness (cost) function of the population. This enables 
improvement to occur [2], and corresponds to the fitness of 
an organism in the natural world [27]. Recombination and 
mutation create the necessary diversity and thereby facilitate 
novelty, while selection acts as a force increasing quality. 
Changed pieces of information resulting from recombination 
and mutation are randomly chosen. Selection operators can be 
either deterministic, or stochastic. In the latter case, individuals 
with a higher fitness have a higher chance to be selected than 
individuals with a lower fitness [2]. 

B. Evolving Agent Populations 

While the construction of Multi- Agent Systems that make 
use of Evolutionary Computing differs [47], [7], they uni- 
versally include populations of agents evolving to provide 
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desired functionality, i.e. evolving agent populations. So, 
extending Chli-De Wilde stability to the class of Multi- Agent 
Systems that make use of Evolutionary Computing requires 
understanding population dynamics and macro- states. 

1 ) Population Dynamics: Change of notation. Section III 
has defined the main properties of Markov chains that we use 
to model Multi- Agent Systems. We have used the standard 
notation, as used in [37] and many other textbooks on 
stochastic processes. This notion uses integers to denote states. 
These integers are easily used to label the rows and colunms 
of the transition matrix. At the end of Section II, we made 
the step from Markov chain to multi-agent system. There are 
many more states than agents, and both ought to be labeled by 
integers. This turns out to be confusing, and therefore we make 
a change of notation here. We will now label the agents with 
integers i, and the states by vectors. Each agent i is in a scalar 
state All n agents together are in an n-dimensional vector 
state |. (An agent can be described by multiple numbers, but 
we group these here into a single scalar just as multiple 
binary digits can be read as a single integer.) 

The states of the agents are random variables. The actual 
values of the random variable will be denoted by numbers 
Xi or Yi. The actual values of the vector of random variables 
^ will be denoted by vectors X or Y. All the states can vary 
in time, and time will be denoted by the superscript t. The 
symbol for t is the same as in Section II, but the symbols i 
and X have a new meaning. The symbol i now denotes the 
agent, and Xi is a number describing the state of the agent i. 

The change of notation is essential because each agent is 
described by a random variable that has the Markov property. 
The multi-agent system is a network of interacting Markov 
chains. Without our change of notation, one would need 
multiple subscripts counting different things. 

A Multi-Agent System that makes use of Evolutionary 
Computing, an evolving agent population, is composed of 
n agents, with each agent i in a state ^- at time t, where 
i = 1, 2, . . . , n. The states of the agents are random variables, 
and so the state vector for the Multi-Agent System is a 
vector of random variables with the time being discrete, 
t = 0, 1, . . . . The interactions among the agents are noisy, and 
are given by the probability distributions 

Pr(Xi|Y)=Pr(4*+i=Xi|^*=Y), i = l,...,n, (7) 

where Xi is a value for the state of agent i, and Y is a value for 
the state vector of the Multi-Agent System. The probabilities 
implement a Markov process [51], with the noise caused by 
mutations. Furthermore, the agents are individually subjected 
to a selection pressure from the environment of the system, 
which is applied equally to all the agents of the population. 
So, the probability distributions are statistically independent, 
and 

Pr(X|Y) = n,Li Pr(C*+' = Xi\e = Y). (8) 

If the occupation probability of state X at time t is denoted 
by then 

Px = Ep^(^|Y)Py'- (9) 

Y 



This is a discrete time equation used to calculate the evolution 
of the state occupation probabilities from t = 0, while equation 
(8) is the probability of moving from one state to another. The 
Multi- Agent System is self- stabilising if the limit distribution 
of the occupation probabilities exists and is non-uniform, i.e. 

= limt^ooPk (10) 
exists for all states X, and there are states X and Y such that 

PxT^Py- (11) 

These equations imply that some configurations of the system 
after an extended time will be more likely than others, because 
the likelihood of their occurrence no longer changes. Such a 
system is stable, because the likelihood of states occurring 
no longer changes with time, and this is the definition of 
stability developed in [15]. This is stability in the stochastic 
sense, because there is some indeterminacy in the future 
evolution described by the probability distributions. So, even 
with knowing the initial conditions, there are many directions 
in which the process might evolve, but still some paths will be 
more probable than others. Equation (10) is the probabilistic 
equivalent of an attractor"^ in a system with deterministic 
interactions, which we had to extend to a stochastic process 
because mutation is inherent in Evolutionary Computing. 

While the number of agents in the Chli-DeWilde formalism 
can vary, we require it to vary according to the selection 
pressure acting upon the evolving agent population. We must 
therefore formally define and extend the definition of dead 
agents, by introducing a new state d for each agent. If an 
agent is in this state, ^\ = d, then it is dead and does not 
affect the state of other agents in the population. If an agent 
i has low fitness then that agent will likely die, because 

Pr(c^|Y) = Pr(^,^+i = d\e = Y) (12) 

will be high for all Y. Conversely, if an agent has high fitness, 
then it will likely replicate, becoming a similarly successful 
agent (mutant), or crossover might occur changing the state of 
the successful agent and another agent. 

2) Population Macro-States: The state of the system, an 
evolving agent population, S is determined by the collection 
of agents of which it consists at a specific time t, which 
potentially changes as the time increases, t+1. This collection 
of agents will have varying fitness values, and so the one with 
the highest fitness at the current time t is the current maximum 
fitness individual. For example, an evolving agent population 
with individuals ranging in fitness between 36.2% and 45.8%, 
the current maximum fitness individual (agent) is the one with 
a fitness of 45.8%. So, we can define a macro-state M as 
a set of states (evolving agent populations) with a common 
property, here possessing at least one copy of the current 
maximum fitness individual. Therefore, by its definition, each 
macro- state M must also have a maximal state composed 
entirely of copies of the current maximum fitness individual. 
If the population size is not fixed (not in nature, can be in 
evolutionary computing), the state space of the evolving agent 

^An attractor is a set of states, invariant under the dynamics, towards which 
neighbouring states asymptotically approach during evolution [55]. 
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maximum macro-state M. 




Fig. 1. State-Space of an Evolving Agent Population: A possible 
evolutionary path through the state-space is shown, with the selection pressure 
of the evolutionary process driving it towards the maximal state of the 
maximum macro-state Mmax- 



population is infinite, but in practise would be bounded by 
resource availability. So, there is also an infinite number of 
configurations for an evolving agent population that has the 
same current maximum fitness individual. 

So, the state-space / of the system (evolving agent pop- 
ulation) S can be grouped to a set macro-states {M}. For 
one macro-state, which we will call the maximum macro- 
state Mjnax^ the current maximum fitness individual will be 
the global maximum fitness individual, which is the optimal 
solution (fittest individual) that the evolutionary computing 
process can reach through the evolving agent population 
(system) S. For example, an evolving agent population at 
its maximum macro-state M^ax, with individuals ranging in 
fitness between 88.8% and 96.8%, the global maximum fitness 
individual (agent) is the one with a fitness of 96.8% and there 
will be no fitter agent. Also, we can therefore refer to all other 
macro-states of the system S as sub-optimal macro-states, as 
there can be only one maximum macro-state Mmax- 

We can consider the macro-states of an evolving agent 
population visually through the representation of the state- 
space / of the system S shown in Figure 1, which includes a 
possible evolutionary path through the state-space. Traversal 
through the state-space / is directed by the selection pressure 
of the evolutionary process acting upon the population S, 
driving it towards the maximal state of the maximum macro- 
state Mynax^ consisting entirely of copies of the optimal 
solution. It is the equilibrium state that the system S is forever 
falling towards without ever quite reaching, because of the 
noise (mutation) within the system. Yet, the maximum macro- 
state Mmax^ in which this maximal state is located, will be 
reached, provided the system does not get trapped at local 
optima, i.e. the probability of being in the maximum macro- 
state Mmax at infinite time is one, ^M^ax ~ 

Furthermore, we can define quantitatively the probability 
distribution of the macro-states that the system occupies at 
infinite time. For a stable system, as defined by equation (11), 
the degree of instability, 5, is the entropy of its probability 
distribution at infinite time, 

5 = H{p°°) = - ^P^Zo5n(p^), (13) 

X 



where N is the number of possible states, and taking log to 
the base N normalises the degree of instability. The degree 
of instability will range between zero (inclusive) and one 
(exclusive), because a maximum instability of one would 
only occur in the theoretical extreme scenario of a non- 
discriminating selection pressure [25]. 

V. Simulation and Results 

A. Evolutionary Dynamics 

A simulated agent population was evolved relative to an arti- 
ficial selection pressure created by a fitness function generated 
from a user request R. An individual (agent) of the population 
consisted of a set of attributes, ai,a2, and a user request 
consisted of a set of required attributes, ri,r2, .... The fitness 
function for evaluating an individual agent A, relative to a user 
request R, was 

fitness{A, R) = . ^ , r, (14) 

where a is an attribute of the agent A such that the differ- 
ence to the required attribute r of R was minimised. The 
abstract agent descriptions was based on existing and emerg- 
ing technologies for semantically capable Service-Oriented 
Architectures [41], such as the OWL-S semantic markup 
for web services [31]. We simulated an agent's semantic 
description with an abstract representation consisting of a 
set of attributes, to simulate the properties of a semantic 
description. Each attribute representing a property of the se- 
mantic description, ranging between one and a hundred. Each 
simulated agent was initialised with a semantic description of 
between three and six attributes, which would then evolve in 
number and content. 

Equation 14 was used to assign fitness values between 0.0 
and 1.0 to each individual of the current generation of the 
population, directly affecting their ability to replicate into the 
next generation. The Evolutionary Computing process was 
encoded with a low mutation rate, a fixed selection pressure 
and a non-trapping fitness function (i.e. did not get trapped at 
local optima^). 

The type of selection used was fitness-proportional and non- 
elitist, with fitness-proportional meaning that the fitter the 
individual the higher its probability of surviving to the next 
generation. Non-elitist means that the best individual from 
one generation was not guaranteed to survive to the next 
generation; it had a high probability of surviving into the next 
generation, but it was not guaranteed as it might have been 
mutated [21]. These initial parameters where chosen to focus 

^ These constraints can be considered in abstract using the metaphor of 
the fitness landscape, in which individuals are represented as solutions to 
the problem of survival and reproduction [58]. All possible solutions are 
distributed in a space whose dimensions are the possible properties of 
individuals. An additional dimension, height, indicates the relative fitness (in 
terms of survival and reproduction) of each solution. The fitness landscape is 
envisaged as a rugged, multidimensional landscape of hills, mountains, and 
valleys, because individuals with certain sets of properties are fitter than others 
[58]. Here the ruggedness of the fitness landscape is not so severe, relative 
to the population diversity (population size and mutation rate), to prevent 
the evolving population from progressing to the global optima of the fitness 
landscape. 



100 



« 60 

CD 

E 40 



20 








Maximum Fitness 
Average Fitness 



global maximum fitness, 



100 Generation 200 



300 



Fig. 2. Graph of Evolutionary Dynamics: This shows both the maximum 
and average fitness increasing over the generations of a typical evolving agent 
population, and as expected the average fitness remains below the maximum 
fitness because of variation in the evolving agent population [23]. 

on studying our extended Chli-DeWilde stability, rather than 
the evolutionary computing process itself. 

Crossover (recombination) was then applied to a randomly 
chosen 10% of the surviving population. Mutations were then 
applied to a randomly chosen 10% of the surviving population; 
point mutations were randomly located, consisting of inser- 
tions (an attribute was inserted into an agent), replacements 
(an attribute was replaced in an agent), and deletions (an 
attribute was deleted from an agent). Rates of 10% where 
chosen, because they would provide the necessary behaviour to 
a sufficient degree. A dynamic population size was used, with 
an initial population size of 300, to ensure exploration of the 
available agent attribute combination space, which increased 
with the average size of the population's agents, because 
fixed population sizes can sometimes fail to sufficient search 
available combination spaces. 

The issue of bloat [26] was controlled by augmenting 
the fitness function with a parsimony pressure [48], which 
biased the search to smaller agents, evaluating larger than 
average agents with a reduced fitness, and thereby providing 
a dynamic control limit which adapted to the average size of 
the individuals of the evolving agent population. 

We first plotted the fitness of the evolutionary process for 
a typical evolving agent population to elucidate its inherent 
evolutionary dynamics. The graph in Figure 2 shows both the 
maximum and average fitness increasing over the generations 
of a typical evolving agent population, and as expected the 
average fitness remains below the maximum fitness because of 
variation in the evolving agent population [23], showing that 
the inherent dynamism of evolutionary processes applies to 
evolving agent populations. 

B. Stability 

1) Initial Parameters: An evolving agent population was 
called stable if the distribution of the limit probabilities existed 
and was non-uniform, as defined by equations (10) and (11). 
The simplest case was a typical evolving agent population with 
a global optimum, which was stable if there were at least two 
macro-states with different limit occupation probabilities. So, 
we considered the maximum macro-state Mmax and one of 
the sub-optimal macro-states, M^ai f • Where the states of the 
macro-state Mmax each possessed at least one individual with 



= lirrit- 



= 1, 



while the states of the macro-state M^aif each possessed at 
least one individual with fitness equal to half of the global 
maximum fitness. 



0, 



thereby fulfilling the requirements of equations (10) and 
(11). A value of t = 1000 was chosen to represent t = 
oo experimentally, because the simulation has often been 
observed to reach the maximum macro-state M^ax within 
500 generations. Therefore, the probability of the system S 
being in the maximum macro-state Mmax at the thousandth 
generation is expected to be one, PM^ax ~ ^' Furthermore, 
the probability of the system being in the sub-optimal macro- 
state Mhaif at the thousandth generation is expected to be 
zero, pi^oo = 0. 

2) Predictions: The sub-optimal macro-state Mhaif, having 
a lower fitness, is predicted to be seen earlier in the evolu- 
tionary process before disappearing as higher fitness macro- 
states are reached. The system S will take longer to reach 
the maximum macro-state Mmax^ but once it does will likely 
remain, leaving only briefly depending on the strength of the 
mutation rate, as the selection pressure was non-elitist. 

3) Results: Figure 3 shows, for a typical evolving agent 
population, a graph of the probability as defined by equation 
(9) of the maximum macro-state Mmax and the sub-optimal 
macro-state M^ai / at each generation, averaged from ten thou- 
sand simulation runs for statistical significance. The behaviour 
of the simulated system S was as expected, being in the 
maximum macro-state Mmax only after generation 178 and 
always after generation 482. It was also observed being in the 
sub-optimal macro-state Mhaif only between generations 37 
and 113, with a maximum probability of 0.053 at generation 
61. This is because the evolutionary path (state transitions) 
could avoid visiting the macro- state. 

4) Conclusions: As expected the probability of being in 
the maximum macro-state Mmax at the thousandth generation 
was one, p]^^^ = 1, and so the probability of being in 




400 600 
Generation 

Fig. 3. Graph of the Probabilities of the Macro-States Mmax and M^aif 
at each Generation: The system S, a typical evolving agent population, was in 
the maximum macro-state Mmax only after generation 178 and always after 
generation 482. It was also observed being in the sub-optimal macro-state 
Mhaif only between generations 37 and 113. 
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any other macro-state, including the sub-optimal macro-state 
Mhaif, at the thousandth generation was zero, Pm^^j^ = 0. 
We can therefore conclude that our extended Chli-DeWilde 
stability accurately models the stability over time of evolving 
agent populations (Multi-Agent Systems with Evolutionary 
Computing). 

C. Visualisation 

1 ) Results: A visualisation for the state of a typical evolv- 
ing agent population, from the experiment of previous subsec- 
tion, at the thousandth generation is shown in Figure 4, with 
each line representing an agent and each shade representing 
an agent attribute, with the identical agents grouped for 
clarity. It shows that the evolving agent population reached 
the maximum macro-state Mmax and remained there, but as 
expected never reached its maximal state, where all the agents 
are identical and have maximum fitness, which is indicated by 
the lack of total uniformity in the visualisation. 




Agent 



Fig. 4. Visualisation of an Evolving Agent Population at the 1000th 
Generation: The population consists of 323 agents, with each line representing 
an agent, and each shade representing an agent attribute. So, we see a 
population consisting of multiple agents, many of which are identical, having 
the same maximum global fitness. The identical agents were grouped for 
clarity, and as expected the system S reached the maximum macro-state 

Mmax • 

2) Conclusions: The result was as expected, a lack of total 
uniformity in the visualisation, because of the mutation (noise) 
within the evolutionary process, which is necessary to create 
the opportunity to find fitter (better) sequences and potentially 
avoid getting trapped at any local optima that may be present. 
We can therefore conclude that the macro- state interpretation 
of our extended Chli-DeWilde stability accurately models 
the state-space of evolving agent populations (Multi-Agent 
Systems with Evolutionary Computing). 

D. Degree of Instability 

1) Results: Given that our simulated evolving agent pop- 
ulation is stable as defined by equations (10) and (11), we 
can determine the degree of instability as defined by equation 
(13). So, calculated from its limit probabilities, the degree of 
instability was 

X 

= -l/o^Ar(l) 
= 0, 



where t = 1000 was an effective estimate for t = oo. This 
result was expected because the maximum macro-state M^ax 
at the thousandth generation was one, plSi^^ = 1, and so the 
probability of being in any other macro- state at the thousandth 
generation was zero. 

2) Conclusions: The system therefore showed no instabil- 
ity, as there is no entropy in the occupied macro- states at 
infinite time. We can therefore conclude that the degree of 
instability of our extended Chli-DeWilde stability can provide 
a macroscopic value to characterise the level of stability 
of evolving agent populations (Multi-Agent Systems with 
Evolutionary Computing). 

E. Stability Analysis 

1 ) Initial Parameters: We then performed a stability analy- 
sis (similar to a sensitivity analysis [13]) of a typical evolving 
agent population, by varying its key parameters while mea- 
suring its stability. We varied the mutation and crossover rates 
from 0% to 100% in 10% increments to provide a sufficient 
density of measurements to identify any trends that might be 
present, calculating the degree of instability, S from (13), at 
the thousandth generation. These degree of instability values 
were averaged over 10 000 simulation runs to ensure statistical 
significance, and graphed against the mutation and crossover 
rates in Figure 5. 

2 ) Results: It shows that the crossover rate had little effect 
on the stability of the evolving agent population, whereas 
the mutation rate did significantly affect stability. With the 
mutation rate under or equal to 60%, the evolving agent 
population showed no instability, with S values equal to zero as 
the system S was always in the same macro-state M at infinite 
time, independent of the crossover rate. With the mutation 
rate above 60% the instability increased significantly, with the 
system being in one of several different macro- states at infinite 
time; with a mutation rate of 70% the system was still very 
stable, having low S values ranging between 0.08 and 0.16, but 




Fig. 5. Graph of Stability with Different Mutation and Crossover Rates: 
With the mutation rate under or equal to 60%, the evolving agent population 
showed no instability, with 6 values equal to zero as the system S was always 
in the same macro-state M at infinite time, independent of the crossover rate. 
With the mutation rate above 60% the instability increased significantly. 
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once the mutation rate was 80% or greater the system became 
quite unstable, shown by high S values nearing 0.5. 

3) Conclusions: As one would have expected, an extremely 
high mutation rate had a destabilising affect on the sta- 
bility of evolving agent populations. Also, as expected the 
crossover rate had only a minimal effect, because variation 
from crossover was limited once the population matured, 
consisting of agents identical or very similar to one another. It 
should also be noted that the stability of the system is different 
to its performance at optimising, because while showing 
no instability with mutation rates below 60% (inclusive), 
it reached the maximum macro-state M^ax only with a 
mutation rate of 10% or above, while at 0% it was stable 
at sub-optimal macro-states between Mtenth (inclusive) and 
^twentieth (inclusivc), i.e. with at least one individual with 
a fitness between twentieth (inclusive) to a tenth (inclusive) 
of the global maximum fitness, as indicated by all of the 0% 
Mutation Rate experiments have degree of instability, 5, values 
of zero in Figure 5. This is because there was no mutation 
(mutate rate = 0%), and so the evolving agent populations 
always remained near the sub-optimal macro-state to which 
they where initialised (seeded), with any crossover rate having 
Httle effect. 

We can therefore conclude that the degree of instability 
of our extended Chli-DeWilde stability can used of perform 
stability analyses (similar to a sensitivity analyses [13]) of 
evolving agent populations (Multi- Agent Systems with Evolu- 
tionary Computing). 

VI. Conclusions 

Our extension of Chli-DeWilde stability was developed 
to provide a greater understanding of stability in Multi- 
Agent Systems that make use of Evolutionary Computing, 
i.e. evolving agent populations. We then built upon this 
to construct an entropy-based definition for the degree of 
instability, which provides information about the level of 
stability, applicable to Multi-Agent Systems with or without 
Evolutionary Computing. Furthermore, it can be used to 
perform a stability analysis, similar to a sensitivity analysis, 
of Multi- Agent Systems. 

Collectively, the experimental results confirm that Chli- 
DeWilde stability has been successfully extended to evolving 
agent populations, while our definition for the degree of insta- 
bility provides a macroscopic value to characterise the level of 
stability. These findings also support the proposition that Chli- 
DeWilde stability can be widely applied to different classes of 
Multi- Agent Systems. So, our extended Chli-DeWilde stability 
is a useful tool for analysing Multi- Agent Systems, with 
or without Evolutionary Computing, providing an effective 
understanding and quantification to help better understand the 
stability of such systems. 

Overall, an insight has been achieved into the stability of 
Multi- Agent Systems that make use of Evolutionary Comput- 
ing, which is a first step in being able to control such systems. 
For example, say one wanted to avoid a number of bad states. 
If the probability of being in those states kept changing with 
time, it would be difficult to devise a strategy to avoid these 



states. However, if the probability converges in time, while one 
could not guarantee to avoid those states, one could at least 
calculate the expected damage, i.e. the probability of being in 
a state times by the penalty for being in it, summed over all the 
states one wishes to avoid. Stochastic control of multi-agent 
systems will be the subject of further work. 

Our future work will also consider include more experi- 
mental scenarios to further consolidate the conclusions, with 
a range of different ^^n^^^ landscapes [58], including flat ones 
(from the neutral theory of molecular evolution [25]) and ones 
with multiple global optima. 
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